Method for interconnecting and system of interconnected processing elements by controlling network density

ABSTRACT

Computer elements in a massively parallel computer system are interconnected in such a way that the number of connections per element can be balanced against the network diameter or worst case path length. This is done by creating a topology that maintains topological properties of hypercubes yet improves flexibility by enumerating the nodes of the network in number systems whose base can be varied. Topologies are generated in which nodes are not always connected when their addresses differ in a single digit. A new variable d is introduced, the purpose of which is to control the overall density of the network by controlling the number of intermediate arc connections within the rings of the network.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to the interconnection ofprocessing elements to form massively parallel computer systems and,more particularly, to a method of interconnecting an unlimited number ofprocessing elements in such a way that the number of connections perelement can be balanced against the network diameter. The inventionspecifically provides non-binary hypercube networks.

2. Description of the Prior Art

As the cost of computer elements declines, the possibility of buildingvery large networks is being realized. Ralph Duncan in "A Survey ofParallel Computer Architectures", Computer, February 1990, pp. 5 to 16,reviews alternative approaches to parallel processing. One class ofparallel architecture described by Duncan involves multiple processorsautonomously executing diverse instructions on diverse data, whichDuncan classifies as MIMD for multiple instruction, multiple datastreams. Within this class of parallel architecture are the ring, mesh,tree, and hypercube topologies. The ring topology is characterized ashaving a communication diameter of N/2 where N is the number of nodes.The communication diameter is defined as the worst case path length. Incontrast, a two-dimensional mesh topology typically has a communicationdiameter of 2(N-1) where, again, N is the number of nodes, and this isalso the typical communication diameter of tree topologies, although inboth cases strategies have been employed to reduce the communicationdiameters.

Of these known network topologies, the hypercube has some interestingtopological properties. A Boolean n-cube or "hypercube" topology usesN=2" processors arranged in an n-dimensional array. Each node, orprocessor, in the array has n=log₂ N bidirectional links to adjacentnodes. FIG. 1 illustrates hypercubes for n=2,3,4. Examples of existinghypercube machines include Caltech's CosmicCube, Intel's iPSC/2 andNCube's Ncube/10. Descriptions of these and other hypercube computersystems are provided by John P. Hayes et al. in "HypercubeSupercomputers", Proceedings of the IEEE, vol. 77, no. 12, December1989, pp. 1829 to 1841. Ophir Frieder in "Multiprocessor Algorithms forRelational-Database Operations on Hypercube Systems", Computer, November1990, pp. 13 to 28, provides a tutorial on hypercube systems,particularly as database engines. Hayes et al. list a variety ofrepresentative applications for hypercube computers.

With specific reference to FIG. 1, note the numbering convention for theaddresses of the nodes. For n=2, there are four nodes (N=2²) withaddresses 00, 10, 01, and 11. Each node is connected to two other nodes,and the rule applied is that each node is connected to an adjacent nodehaving an address which differs by only one bit. For n=3, there areeight nodes (N=2³) with addresses 000, 001, 010, 011, 100, 101, 110, and111, but the connection scheme is the same with, for example, node 000being connected to nodes 001, 010 and 100. There are sixteen nodes forthe case where n=4 (N=2⁴), but still the same connection scheme isemployed. Laxmi N. Bhuyan and Dharma P. Agrawal in "GeneralizedHypercube and Hyperbus Structures for a Computer Network", IEEETransactions on Computers, vol. c-33, no. 4, April 1984, pp. 323-333,describe a generalization of the binary hypercube to a variable radixnumbering. In this scheme, the nodes of a network are numbered in asystem in which each digit of the number can be based on a differentradix. Nodes are connected whenever the resulting node addresses differin exactly one digit.

While hypercube architectures offer some very attractive topologicalproperties, as these networks go beyond a few thousand nodes, thenumbers of connections per node and path lengths grow in fixed ways. Ifeither connections per node or path lengths become too large for a giventechnology, the designer is forced to develop new network algorithmsfrom scratch. These new ad hoc topologies may then adversely effect theperformance of existing programs that ran well on smaller networks witholder topologies.

SUMMARY OF THE INVENTION

It is therefore an object of the present invention to a provide a methodof interconnecting processors to form a generalized hypercube structurethat supports an unlimited number of nodes.

It is another object of the invention to provide a computer structureformed of interconnected processors wherein the number ofinterconnections per processor is adjusted against the network diameter.

According to the invention, there is provided a method forinterconnecting processing elements in such a way that the number ofconnections per element can be balanced against the network diameter.This is done by creating a topology that maintains many of the wellknown and desirable topological properties of hypercubes while improvingflexibility by enumerating the nodes of the network in number systemswhose base can be varied. When using a base two number system, thismethod creates the familiar Boolean binary hypercube topology. However,in contrast to a Boolean binary hypercube, the practice of the inventionresults in a non-binary hypercube having fewer interconnectionspermitting the practical realization of very large computer systemshaving an unlimited number of nodes.

The subject invention is a refinement of the scheme described by LaxmiN. Bhuyan and Dharma P. Agrawal, supra. The invention generatestopologies in which nodes are not always connected when their addressesdiffer in a single digit. The invention introduces a new variable d, thepurpose of which is to control the overall density of the network bycontrolling the number of intermediate arc connections within the ringsof the network. For a network having N nodes, the assumption is madethat there exist positive integers b and n such that N=b^(n). The newvariable d is picked such that 1≦d≦b/2, if b is even, and 1≦d≦(b-1)/2,if b is odd. The nodes of the network are numbered to the base b. Twonodes, x and y, of the network are connected if and only if

1. the address of x differs from the address of y in exactly one digit,and

2. for the digit i where x_(i) ≠y_(i), y_(i) =(x_(i) +j) mod b or y_(i)=(x_(i) -j) mod b for some j≦d.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, aspects and advantages will be betterunderstood from the following detailed description of a preferredembodiment of the invention with reference to the drawings, in which:

FIG. 1 is a pictorial representation of some exemplary Boolean hypercubetopologies;

FIG. 2 shows the topology of a typical ring in a base 8 network whered=1;

FIG. 3 shows the topology of a typical ring in a base 8 network whered=2;

FIG. 4 shows the topology of a typical ring in a base 8 network whered=3;

FIG. 5 shows the topology of a typical ring in a base 8 network whered=4;

FIG. 6 is a block diagram of a 125 computer network where b=5, n=3 andd=1;

FIG. 7 is a block diagram of the lowest level of packaging of animplementation of the invention composed of 4096 nodes;

FIG. 8 is a block diagram of the next level of packaging of theembodiment shown in FIG. 7; and

FIG. 9 is a block diagram of the highest level of packaging of theembodiment shown in FIG. 7 of the drawings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS OF THE INVENTION

Before describing a specific implementation of the invention, thenetwork connection algorithm is discussed in general terms. Thealgorithm is based on the assumption that there exists positive integersb and n such that N=b^(n), where N is the number of nodes in thenetwork. An integer d is chosen between one and b/2, if b is even, orbetween one and (b-1)/2, if b is odd. Once the integer d has beenselected, the nodes of the network are numbered to the base b. In otherwords,

N=a_(m) b^(m) +a_(m-1) b^(m-1) +. . .+a₁ b¹ +a₀ b^(n),

where a₀, a₁, . . ., a_(m-1), a_(m) are non-negative integers each lessthan b. For a base ten (b=10) number system, the symbols a_(i) arecalled "digits", and for a base two (b=2) number system, the symbolsa_(i) are called "bits". Without loss of generality, the symbols a_(i)are referred to hereinafter as "digits" even though the base numbersystem may be other than ten.

Next, connect two nodes, x and y, of the network if and only if

1. the address of x differs from the address of y in exactly one digit,and

2. for the digit i where x_(i) ≠y_(i), y_(i) =(x_(i) +j) mod b or y_(i)=(x_(i) -j) mod y for some j≦d.

Notice that -x mod b=(y-x) mod b. For example, if b=5 and d=1, then anode whose address is 1234 would be connected to nodes having addresses2234, 0234, 1334, 1134, 1224, 1244, 1233, and 1230. Notice that dcontrols the density of the network. The larger d becomes, the denserthe network becomes.

During the application of the algorithm, the ports of the various nodesare labeled to indicate both the digit and digit value of the node towhich they connect. For example, a port labeled 3,7 would connect to anode whose third digit is 7. The number of connections per node isμlog_(b) N, where μ=2d, except if b is even and d=b/2 in which caseμ=b-1. This can be seen to be true by considering that for each integerless than or equal to d, there are two connections for each digit of anode address. The number of digits in the node address is log_(b). Thediameter of the network is given by ##EQU1## where ##EQU2## is alwaysrounded up to the next integer when it is not an integer.

Referring now to the drawings, and more particularly to FIG. 2, there isshown the topology of a typical ring in a base 8 (i.e., b=8) networkwhere d=1. There are eight computers or nodes, N, in this network sothat n=1 or N=8¹. While this illustration is trivial, the next threefigures illustrate the effect of increasing the value of d on thedensity of the network. FIG. 3 shows the case for d=2. In this case,each computer connects to four other computers. FIG. 4 shows the casefor d=3. In this case, each computer connects to six other computers.FIG. 5 shows the case for d=4. In this case each computer connects toseven other computers. Thus, FIGS. 2, 3, 4, and 5 provide a graphicillustration of the increasing density of a simple eight node networkfor an increasing value of d.

A more practical application of the interconnection technique accordingto the invention is illustrated in FIG. 6 which shows a 125 computernetwork. In this example, the base, b, is set to equal five so that n=3;that is, N=b^(n) =5³ =125. In order to minimize the density of thisnetwork, d is set equal to one. This network is implemented in fiveplanes 10, 11, 12, 13, and 14, with each plane having twenty-fivecomputers 15. Within each plane there are five rings and five loops.Each computer in the network is connected to six other computers, twocomputers within its ring, two computers within its loop and twocomputers in different planes.

Even for this still relatively simple network, the selection of thevariable d allows a control of the density of the network not known inthe prior art. The network shown in FIG. 6 may be easily realized as apractical embodiment of five printed circuit boards with edgeconnectors. However, the invention is especially useful for very largenetworks where interconnection densities become critical to whether animplementation is realizable in the real world.

To illustrate this, a specific implementation of the invention for d=1,b=8 and n=4 is shown in FIGS. 7, 8 and 9. Thus, for this implementationN=b^(n) =8⁴ =4096. For this very large number of nodes, the densityvariable d is chosen to be one to minimize the density.

FIG. 7 shows the lowest level of packaging. Each box 20₀ to 20₇ containseight computers interconnected with forty-eight (3×16) data paths 22₁,22₂ and 22₃ per box. The complete package 24 in FIG. 7 thus containssixty-four (8×8) computers. The package 24 is interconnected to the restof the computers in the system with 256 (2×128) data paths 26₁ and 26₂.

FIG. 8 shows the next level of packaging. Each package 24₀ to 24₇contains 64 computers with 256 data paths as shown in FIG. 7. The largerpackage 30 thus contains 512 (8×64) computers supporting 1024 (2×512)data paths. FIG. 9 shows the highest level of packaging containing eightpackages 30₀ to 30₇, each containing 512 computers and 1024 data paths.There are 4096 (8×512) computers with 1024 data paths in the highestlevel of packaging.

In this network of 4096 nodes, each node is directly connected to onlyeight other nodes. This permits any node to communicate with any othernode in the network by involving no more than sixteen intermediatenodes. This compares well with the case of the Boolean binary hypercubewhich demands a denser network with each node connected to twelve othernodes rather than eight. Thus, the invention allows for a reduction of33% in wiring density as compared with the Boolean binary hypercube.This reduction, however, is traded for a concomitant 33% increase innetwork diameter.

To route messages in a network according to the invention, first comparethe destination address with the current address. If they are the same,routing has been completed. If not, select some digit, i, for which thecurrent and destination addresses are different. Forward the message outof the port for this digit that differs from the destination address bythe least amount. Continue this procedure until the current anddestination addresses are equal. Note that the selection of the digit towork on at a particular time can be based on which ports are not busy ordamaged. This is common procedure for hypercubes. A port is said to bedamaged if the port itself is damaged or the port to which it connectsis damaged.

The invention simplifies the interconnection of massively parallelcomputer systems yet preserves the desirable properties of a hypercube.These properties are (1) a large number of alternate paths, (2) veryhigh aggregate bandwidth, and (3) uniform connections. Existing methodscan be used to used to route messages within the network. Generally, theinvention provides a non-binary hypercube with less density therebyallowing the practical realization of massively parallel computersystems having an unlimited number of nodes, but as illustrated in FIG.6, the invention also has practical application for smaller computernetworks.

While the invention has been described in terms of illustrative andpreferred embodiments, those skilled in the art will recognize that theinvention can be practiced with modification within the spirit and scopeof the appended claims.

Having thus described my invention, what I claim as new and desired tosecure by Letters Patent is as follows:
 1. A computer system in a formof a non-binary hypercube comprising:a plurality of nodes, N, whereN=b^(n), b and n being positive integers and b>2, each node having anidentifying address numbered according to a number system base b; meansfor determining an address of each of said plurality of nodes; and meansfor interconnecting each respective node of said plurality of nodes tonodes having addresses differing from the respective node's address byexactly one digit to form a toroidal network, a number of connectionsper node being μlog_(b) N, where μ=2d, 1≦d≦b/2 if b is even and1≦d≦(b-1)2 if b is odd, except if b is even and d=b/2, in which caseμ=b-1, to thereby control density of said toroidal network, wherein twonodes, x and y, of the network are connected if and only if1. an addressof x differs from an address of y in exactly one digit, and
 2. for adigit i where x_(i) ≠y_(i), y_(i) =(x_(i) +j) mod y or y_(i) =(x_(i) +j)mod y for some j≦d.
 2. The computer system recited in claim 1 whereinb=8, n-1, and d=2.
 3. The computer system recited in claim 1 whereinb=8, n=1, and d=3.
 4. The computer system recited in claim 1 whereinb=8, n=1, and d=4.
 5. The computer system recited in claim 1 whereinb=5, n=3, and d=1.
 6. The computer system recited in claim 1 whereinb=8, n=4, and d=1.
 7. A computer implemented method of interconnecting aplurality of nodes in a network comprising steps of:providing aplurality of nodes; defining b and n such that a number N of saidplurality of nodes equals b^(n) and b and n are positive integers andb>2; choosing a positive integer d such that 1≦d≦b/2 if b is even and1≦d≦(b-1)/2 if b is odd; defining an address for respective ones of eachof the plurality of nodes by numbering them using numbers represented bya base b numbering system; determining an address of each of saidplurality of nodes; and connecting two nodes of said plurality of nodes,to thereby form a toroidal network, if and only if an address of a firstone of said two nodes of said plurality of nodes differs from an addressof a second one of said two nodes of said plurality of nodes in exactlyone digit, a i^(th) digit and, for a unique i^(th) digit of theaddresses of said two nodes, symbolized by x_(i) and y_(i),respectively, where x_(i) ≠y_(i), y_(i) =(x_(i) +j) mod y or y_(i)=(x_(i) +j) mod y for some j≦d to thereby control density of saidtoroidal network.